There are many factors that affect student learning — but to what extent do school districts themselves actually contribute?

That was the question at the center of a new state audit presented to lawmakers this week that ranked Utah school districts according to their “effectiveness.”

In order to do that, researchers took test scores and adjusted for difference in student demographics — like race and income — that they say are statistically associated with test performance.

Its results suggest that some districts with lower average test scores may be doing a better job advancing student achievement than districts that report higher test scores on average.

For instance, although Park City School District students generally outperform Salt Lake City School District students on state tests, the audit found that Salt Lake City is more effective when it comes to improving student performance, after accounting for inherent differences in the districts’ student populations.

This means despite serving a higher percentage of minority and low-income students, Salt Lake City School District has produced higher learning gains than researchers expected relative to those that would be expected in Park City, which has fewer minority and low-income students.

Still, though, districts’ overall value add is relatively minor, the audit concludes.

“Student demographics and attributes play the largest role in explaining the differences in student achievement,” the 38-page audit states. “While district practices have an effect on student performance, it may be smaller than people believe.”

The audit does not attempt to answer why achievement gaps exist between white and Black, Hispanic and other student groups. Rather, it cites a substantial body of research on the topic has shown minority and low-income students underperform compared to their white, non-economically disadvantaged peers.

“We’re not saying anything about why that’s the case,” said Sven Wilson, co-founder of Notalys, the firm contracted by state auditors to perform the study. “Given American society, we have a lot of disparities that are correlated with race, and one of them is educational performance.”

But one education policy expert said he believes the study’s findings are the equivalent of determining “water is wet.”

“I appreciate, and I think it’s very good research to account for some of these demographic differences,” said Travis Bristol, an associate professor who studies education equity, race and gender, urban schooling and more at University of California, Berkeley’s School of Education.

“But I think that, to me,” he continued, “the larger question is: ‘Are they still doing well enough?’”

How did researchers conduct the audit?

The study relied on Utah testing data from grades one through 10 for two specific time ranges: 2017-2019 and 2021-2023.

Researchers then “restandardized” that data in order to estimate how students performed relative to other students who took the same tests at the same time, according to the report.

Then, they compared those scores across districts and ranked them by “district effect,” which was meant to capture the value added, if any, relative to other districts in the state.

The ranking process involved two different analyses, researchers explained.

The first adjusted for student income and race/ethnicity. Controlling for those, Wilson said, helped researchers “capture how much better or worse districts do than what their demographics would predict.”

The second ranking did not adjust for those factors, relying only on raw test scores.

Unadjusted, the school districts that ranked among the most impactful on student learning included the North Summit, Cache County, Rich, Morgan and Park City school districts.

But adjusted, the rankings changed, with the Salt Lake City, Daggett, and Wayne school districts joining Rich and North Summit in the top five spots.

Ogden City School District ranked last in both analyses.

“Salt Lake has demographics that predict much lower scores than we actually observe,” Wilson said. “Which is why the ‘district effect’ of that district is so high — they significantly outperform expectations.”

Districts themselves tend to have a small impact, audit finds

While there were noticeable changes after adjusting for student demographics, researchers found that district practices accounted for only about 11% of all factors contributing to performance.

Auditors attributed another 23.4% to variation across testing events, differences across grade levels and general randomness, according to the audit.

But, auditors concluded, 65.5% of the differences in student test scores were due to differences in the students themselves."

“Unfortunately, we cannot explain these differences,” Wilson said. He added that race and socioeconomic status play “some role” but “most” of the contributing factors were not included in the data they analyzed.

That includes “innate differences,” like differences in family support and teacher impact, the report states. Auditors cited separate research that indicates such differences contribute to student performance, the report states

“Since we do not know the classrooms students are in or anything about the teachers they have, we cannot determine the effects of teachers,” according to the audit.

Reframing the context

Bristol argued that if Utah lawmakers wish to move the needle for “Black and brown students,” the research must look beyond demographics and interrogate the “social context,” or environment, in which students are learning and living.

“Their job, as policymakers, is to improve the lives of the people that they serve, and you can’t change a person’s race,” Bristol said.

He added that overemphasizing race in such fatalistic terms “abdicates” state and school leaders from their responsibility.

“To lead with race, recognizing that that isn’t something that you can change, means that the policymaker can focus more on their constituents in Park City, because they can’t change the race of the people in Salt Lake City,” Bristol argued.

The audit’s aim, Wilson argued, was to examine districts’ value-add outside of factors they cannot control, which includes students’ race and income, something the report repeatedly notes. Researchers intended to provide an estimation of that impact, Wilson said.

“This study isn’t designed to come out with a list of ‘Top 10 things that school districts can do,’” Wilson said.

To understand why the Salt Lake City District seems to outperform expectations, more research would be needed, he said.

Still, Salt Lake City’s minority and low-income students are trailing behind their white peers in nearly every subject area, according to Utah State Board of Education data.

“That’s why the discussion has to be about the context,” Bristol said. “What are some of the other resources needed to allow the students in Salt Lake to be able to participate in the same Utah economy that the students in Park City are being prepared to participate in and thrive in?”

He noted that highly qualified teachers are a big part of that discussion, as well as wraparound services — all factors within policymakers’ control.

“What you can change is the qualifications of the teachers,” Bristol said. “You can change class size. The resources that the school receives. The amount of social workers that the schools receive.”

The audit separately found that smaller student-teacher ratios are not correlated with better outcomes, instead reporting “the larger the class size, the higher performance.”

Bristol questioned those findings, reiterating that teacher qualifications matter when it comes to class sizes.

“You could have an unqualified teacher teaching a class of five students versus a qualified teacher teaching a class of 20 students, and the outcomes would be different,” Bristol said.

Wilson agreed.

“Teachers matter an awful lot,” he said, adding that a similar analysis at the school level, not district level, would likely yield different results.

“All children are capable of learning,” Wilson said.